{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.011184,
     "end_time": "2021-02-28T17:17:25.882205",
     "exception": false,
     "start_time": "2021-02-28T17:17:25.871021",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Application: Heterogeneous Effect of Sex on Wage Using Double Lasso\n",
    "\n",
    " We use US census data from the year 2012 to analyse the effect of gender and interaction effects of other variables with gender on wage jointly. The dependent variable is the logarithm of the wage, the target variable is *female* (in combination with other variables). All other variables denote some other socio-economic characteristics, e.g. marital status, education, and experience.  For a detailed description of the variables we refer to the help page.\n",
    "\n",
    "\n",
    "\n",
    "This analysis allows a closer look how discrimination according to gender is related to other socio-economic variables.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": 0.39942,
     "end_time": "2021-02-28T17:17:26.291369",
     "exception": false,
     "start_time": "2021-02-28T17:17:25.891949",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "library(hdm)\n",
    "data(cps2012)\n",
    "str(cps2012)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": 2.894102,
     "end_time": "2021-02-28T17:17:29.196986",
     "exception": false,
     "start_time": "2021-02-28T17:17:26.302884",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# create the model matrix for the covariates\n",
    "X <- model.matrix(~-1 + female + female:(widowed + divorced + separated + nevermarried +\n",
    "hsd08 + hsd911 + hsg + cg + ad + mw + so + we + exp1 + exp2 + exp3) + +(widowed +\n",
    "divorced + separated + nevermarried + hsd08 + hsd911 + hsg + cg + ad + mw + so +\n",
    "we + exp1 + exp2 + exp3)^2, data = cps2012)\n",
    "X <- X[, which(apply(X, 2, var) != 0)] # exclude all constant variables\n",
    "demean<- function (x){ x- mean(x)}\n",
    "X<- apply(X, 2, FUN=demean)\n",
    "dim(X)\n",
    "\n",
    "# target variables, index.gender specifices coefficients we are interested in\n",
    "index.gender <- grep(\"female\", colnames(X))\n",
    "y <- cps2012$lnw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.008565,
     "end_time": "2021-02-28T17:17:29.215145",
     "exception": false,
     "start_time": "2021-02-28T17:17:29.206580",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The parameter estimates for the target parameters, i.e. all coefficients related to gender (i.e. by interaction with other variables) are calculated and summarized by the following commands:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_execution_state": "idle",
    "_uuid": "051d70d956493feee0c6d64651c6a088724dca2a",
    "papermill": {
     "duration": 78.426974,
     "end_time": "2021-02-28T17:18:47.651893",
     "exception": false,
     "start_time": "2021-02-28T17:17:29.224919",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# this cell takes a minute to run\n",
    "\n",
    "effects.female <- rlassoEffects(x = X, y = y, index = index.gender)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": 0.115154,
     "end_time": "2021-02-28T17:18:47.787349",
     "exception": false,
     "start_time": "2021-02-28T17:18:47.672195",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "result=summary(effects.female)\n",
    "result$coef\n",
    "library(xtable)\n",
    "print(xtable(result$coef[,c(1,2,4)], type=\"latex\"), digits=3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.008891,
     "end_time": "2021-02-28T17:18:47.805643",
     "exception": false,
     "start_time": "2021-02-28T17:18:47.796752",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now, we estimate and plot confident intervals, first \"pointwise\" and then the joint confidence intervals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": 1.082155,
     "end_time": "2021-02-28T17:18:48.896774",
     "exception": false,
     "start_time": "2021-02-28T17:18:47.814619",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "pointwise.CI <- confint(effects.female, level = 0.90)\n",
    "pointwise.CI\n",
    "plot(effects.female, level=0.90) # plot of the effects\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.01145,
     "end_time": "2021-02-28T17:18:48.920882",
     "exception": false,
     "start_time": "2021-02-28T17:18:48.909432",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Finally, we compare the pointwise confidence intervals to joint confidence intervals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": 1.021263,
     "end_time": "2021-02-28T17:18:49.953381",
     "exception": false,
     "start_time": "2021-02-28T17:18:48.932118",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "joint.CI <- confint(effects.female, level = 0.90, joint = TRUE)\n",
    "joint.CI\n",
    "plot(effects.female, joint=TRUE, level=0.90) # plot of the effects\n",
    "print(xtable(joint.CI), type=\"latex\")\n",
    "\n",
    "# the plot output does not look great"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 89.365707,
   "end_time": "2021-02-28T17:18:51.003711",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2021-02-28T17:17:21.638004",
   "version": "2.2.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
